August  14,  2003  Blackout 

Final  Report 


U.S. -Canada  Power  System 
Outage  T ask  Force 

Final  Report  Summary 


Overview 


■  The  blackout  and  the  investigation 

■  Findings  -  how  the  blackout  happened 

■  Why  the  blackout  happened 

■  Report  recommendations 
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The  blackout 


August  14,  2003,  starting 
at  16:05:57  EDT 


■  Over  50  million  people 
out  of  power  in 
northeast  US  and 
Ontario,  Canada 

■  61,800  NW  of  load  lost 

■  Millions  of  work  hours 
and  billions  of  $  in 
economic  costs 


The  investigation 


■  Directed  by  President  Bush  and  Prime  Minister 
Chretien  on  8/15/03 

■  Task  Force  headed  by  US  Secretary  of  Energy 
and  Canadian  Minister  of  Natural  Resources,  with 
US  and  Canadian  officials 

■  Three  work  groups  -  Electric  System  Outage, 
Nuclear  and  Security 

■  I  nvestigators  from  both  nations,  public  and 
private  sector 

■  Electric  system  investigation  team  composed  of 
NERC,  industry  experts  across  US  and  Canada, 

US  and  Canada  federal  staff 
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Computer  problems 


At  12:15  EDT,  Ml  SO  began  having  problems 
with  its  state  estimator;  it  didn't  return  to  full 
functionality  until  16:04  EDT. 

Sometime  after  14: 14  EDT,  FirstEnergy  began 
losing  its  energy  management  system  (EMS) 
alarms  but  didn't  know  it. 

At  14:20  EDT,  parts  of  FE's  EMS  began  to  fail  - 
first  remote  sites,  then  core  servers  -  but  FE 
system  operators  didn't  know  and  FE  IT  support 
didn't  tell  them. 

Without  a  functioning  EMS,  FE  operators  didn't 
know  their  system  was  losing  lines  and  voltage 
until  perhaps  15:45  EDT. 


L  Reactive  power  problems 

■  At  13:31  EDT,  FirstEnergy  lost 
the  Eastlake  5  plant,  a  critical 
source  of  real  and  reactive 
power  for  the  Cleveland- Akron 
area. 

■  FE  did  not  perform  contingency 
analysis  after  this  trip. 

■  FE  did  not  have  sufficient 
understanding  of  the  Cleveland- 
Akron  area  to  know  their 
system  was  seriously  deficient 
in  reactive  power  for  voltage 
support. 


Cleveland  area  voltage  began 
falling 
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Hourly  West-East  345kV  Actual  (Measured)  Voltages  on  August  14th 
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Line  outages 


■  Between  15:05  and  15:41 
EDT,  FE  lost  3  345  kV  lines  in 
the  Cleveland- Akron  area 
under  normal  loading  due  to 
contact  with  too- tall  trees  - 
but  didn't  know  it  without  EMS. 

■  Line  loading  shifted  and 
reactive  power  demands 
increased  with  each  line  loss. 
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Area  of  Detail 


■  Between  15:39  and  16:08 
EDT,  FE  lost  16  138kV  lines  in 
the  Cleveland- Akron  area  due 
to  overloads  and  ground  faults. 
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The  Ohio  tipping  point 


At  15: 57: 05  EDT,  FE  lost 


the  Sammis-Star  345  kV 
line  due  to  overload. 

■  This  shut  down  a  major 
path  for  power  imports  into 
the  Cleveland-Akron  area 
and  was  the  starting  point 
for  the  full  cascade. 
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The  cascade 


A  cascade  is  a  dynamic  phenomenon  in  an 
electric  system  that  can't  be  stopped  by 
human  intervention  once  it  starts 

■  Power  swings,  voltage  fluctuations  and 
frequency  fluctuations  cause  sequential  trips 
of  numerous  transmission  lines  and 
generators,  and  automatic  load-shedding  in  a 
widening  geographic  area. 

■  System  oscillations  grow  so  large  the  system 
can't  rebalance  and  stabilize. 
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Map  Legend:  Probable  Cause 

•  Configuration  Isolated  •Under-Voltage  •  Consequential  Tripping 

•  Under-Frequency  •  Operator  Shutdown  •  Not  Determined 


Why  did  Ohio  start  happen? 


The  Ohio  phase  began  because: 

1.  FirstEnergy  and  ECAR  failed  to  study  and 
understand  the  inadequacies  of  the  FE  system 
and  FE  didn't  operate  the  system  with 
appropriate  voltage  criteria. 

2.  FE  had  inadequate  situational  awareness  and 
didn't  recognize  its  system  deteriorating. 
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Why  did  Ohio  start  happen? 


38'  Height  @  5  MPH  Winds 
36’  Height  @  0  MPH  Winds 
34’  Height  @  Emergency  Rating 


800' 


3.  FE  failed  to  trim  trees  in  its  rights  of  way,  so 
each  early  345  kV  line  faulted  on  a  tree  that 
was  too  tall. 

4.  Ml  SO  and  neighboring  PJ  M  didn't  provide 
effective  real-time  diagnostic  support  to  FE. 

5.  FE  didn't  act  to  restore  its  system  to  a  secure 
condition  (due  to  the  other  causes). 
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Why  the  cascade  happened 


Sammis-Star  and  the  Cleveland-Akron  line 
trips  shifted  load  burden  onto  limited  paths 


2.  After  Sammis-Star,  Zone  3  (and  2)  relays  in 
Ohio  and  Michigan  caused  13  line  trips 
between  15:57:05  and  16:10:38.350  EDT  that 
would  not  have  happened  so  quickly  without 
those  relays. 


The  cascade  (2) 


3.  Relay  settings  on  lines,  generators  and 
load-shedding  across  the  northeast 
were  not  coordinated  and  integrated  to 
reduce  the  likelihood  of  a  cascade,  so 
the  grid's  elements  and  regions  couldn't 
rebalance. 

4.  Physics  -  once  the  frequency,  voltage 
and  power  swings  started,  the  grid 
couldn't  recover 


16 


Report  recommendations  for 
.  .better  reliability 


Institutional  issues  (14  recommendations), 
including: 

■  Make  reliability  standards  mandatory  and 
enforceable 

■  Develop  an  independent  funding  mechanism 
for  NERC 

■  Strengthen  reliability  institutions,  including 
NERC  and  regional  councils,  and  define 
minimum  requirements  and  cleaner  footprints 
for  control  areas  and  reliability  authorities 

■  Reliability  investments  should  be  recoverable 
in  transmission  rates 
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.  Reliability  recommendations 

Strengthen  and  support  NERC's  actions  of 
February  14,  2004  (17),  including: 

■  Correct  the  direct  causes  of  the  blackout 

■  Strengthen  NERC  compliance  program 

■  Support  NERC's  Reliability  Readiness  Audits 

■  I  mprove  operator  training  and  certification 

■  Use  better  system  protection  measures 

■  Use  better  real-time  tools  for  grid  monitoring 
and  operation 

■  Accelerate  and  improve  reliability  standards 
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.  Reliability  recommendations 

Physical  and  cyber- security  ( 13) : 

■  Implement  NERC  IT  standards 

■  Develop  IT  management  procedures 

■  Improve  IT  forensic  and  diagnostic 
capabilities 

■  Establish  authority  for  physical  and 
cyber  security 

■  Control  access  to  critical  equipment 
Canadian  nuclear  power  (2) 
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t  _  Will  More  Blackouts  Occur? 


■  System  is  man-made,  and  subject  to 
mechanical  failure  and  human  error. 

■  NERC's  readiness  audits  are  a  key 
preventive  measure. 

■  The  U.S  -  Canada  Task  Force  has  been 
extended  for  a  year  to  provide  oversight 
for  implementation  of  recommendations. 
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